Implementing Shared Mixed Reality

ABSTRACT

A system for implementing a shared mixed reality experience to participants in a same physical room having a plurality of VR headsets, each of which is adapted to be worn by a participant in the room. Each VR headset having a forward-facing color camera stereo pair. The system includes a computer in communication with at least one of the VR headsets. The system includes a memory in communication with the computer. The memory storing an original 3D digital representation of the room. The computer calculates a rendered VR scene for the at least one of the headsets from a point of view of each of the participant&#39;s two eyes, based on the VR headset&#39;s current position and orientation. A method for implementing a shared mixed reality experience to participants in a same physical room.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a nonprovisional of U.S. provisional patent application Ser. No.63/355,889 filed Jun. 27, 2022, incorporated by reference herein.

FIELD OF THE INVENTION

The present invention is related to providing participants in a samephysical room, a shared mixed virtual reality. More specifically, thepresent invention is related to providing participants in a samephysical room, a shared mixed virtual reality utilizing VR headsets,each of which has a forward-facing color camera stereo pair.

BACKGROUND OF THE INVENTION

This section is intended to introduce the reader to various aspects ofthe art that may be related to various aspects of the present invention.The following discussion is intended to provide information tofacilitate a better understanding of the present invention. Accordingly,it should be understood that statements in the following discussion areto be read in this light, and not as admissions of prior art.

When multiple participants share a virtual reality (VR) experience inthe same physical room, they cannot physically see each other. To allowthose participants to see each other, and thereby to create a sociallyshared mixed reality experience, prior work has proceeded in one of twoways. Either (1) participants are shown to each other as syntheticavatars of themselves, or else (2) the shared experience is stagedwithin a prepared room, such as a room with green colored walls, so thatthe VR headsets can use forward facing cameras to continue to see theother participants in the room, while a green screen algorithm is usedto visually replace the surrounding walls with a shared synthetic VRworld.

BRIEF SUMMARY OF THE INVENTION

The present invention pertains to a system for implementing a sharedmixed reality experience to a first participant and a second participantin a same physical room. The system comprises a first VR headset havinga forward-facing color camera stereo pair worn by the first participantin the physical room. The system comprises a second VR headset having aforward-facing color camera stereo pair worn by the second participantin the physical room. The system comprises a computer in communicationwith the first and second VR headsets. The system comprises a memory incommunication with the computer. The memory storing an original 3Ddigital representation of the room. The computer calculates a rendered3D VR scene for the first VR headset from a point of view of the firstparticipant's two eyes, based on the first VR headset's current positionand orientation. Anything in the room that is different from theoriginal 3D digital representation of the room is visible to the firstparticipant in the rendered 3D VR scene. Any part of the room that waspresent in the original 3D digital representation of the room becomesvisibly replaced by the rendered 3D VR scene. The computer calculates arendered 3D VR scene for the second VR headset from a point of view ofthe second participant's two eyes, based on the second VR headset'scurrent position and orientation. Anything in the room that is differentfrom the original 3D digital representation of the room is visible tothe second participant in the rendered 3D VR scene. Any part of the roomthat was present in the original 3D digital representation of the roombecomes visibly replaced by the rendered 3D VR scene.

The present invention pertains to a system for implementing a sharedmixed reality experience to a first participant and a second participantin a same physical room. The system comprises a first VR headset worn bythe first participant in the physical room. The system comprises asecond VR headset worn by the second participant in the physical room.Each VR headset as a forward-facing color camera stereo pair and amemory. The memory storing an original 3D digital representation of thephysical room. The system comprises a first auxiliary processor adjacentto the first participant. The first auxiliary processor in wiredconnection to the first VR headset worn by the first participant. Thesystem comprises a second auxiliary processor worn by the secondparticipant. The second auxiliary processor in wired connection to thesecond VR headset worn by the second participant. The memory of thefirst headset worn by the first participant in communication through thewired connection with the first auxiliary processor worn by the firstparticipant. The memory of the second headset worn by the secondparticipant in communication through the wired connection with thesecond auxiliary processor worn by the second participant. The firstauxiliary processor may be worn on a belt or armband of the firstparticipant, or may be attached to a chair the first participant issitting in or may be positioned under the chair in which the firstparticipant may be sitting. The second auxiliary processor may be wornor positioned the same way by the second participant.

The present invention pertains to a method for implementing a sharedmixed reality experience to participants in a same physical room. Themethod comprises the steps of storing an original 3D digitalrepresentation of the room in a memory. There is the step of calculatingwith a computer a rendered 3d VR scene for at least one VR headset wornby a participant of a plurality of headsets worn by participants in theroom from a point of view of each of the participant's two eyes, basedon the one VR headset's current position and orientation. Anything inthe room that is different from the original 3D digital representationof the room is visible to the participant in the rendered 3D VR scene.Any part of the room that was present in the original 3D digitalrepresentation of the room becomes visibly replaced by the rendered 3DVR scene. Each VR headset has a forward-facing color camera stereo pair.The computer is in communication with at least the one VR headset. Thememory is in communication with the computer. There is the step ofdisplaying in the one VR headset the rendered VR scene.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a representation of the three different embodiments of thepresent invention, all of which emanate from a common scanned 3D modelof a physical room or a synthetic representation of a physical room.

FIG. 2 is a representation of two participants in a physical roomutilizing the first or third embodiments of the present invention.

FIG. 3 is a representation of two participants in the physical roomutilizing the second embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings wherein like reference numerals refer tosimilar or identical parts throughout the several views, and morespecifically to FIG. 1 thereof, there is shown a system 10 forimplementing a shared mixed reality experience to a first participant 16and a second participant 21 in a same physical room 17. The system 10comprises a first VR headset having a forward-facing color camera stereopair 18 worn by the first participant 16 in the physical room 17. Thesystem 10 comprises a second VR headset having a forward-facing colorcamera stereo pair 18 worn by the second participant 21 in the physicalroom 17. The system 10 comprises a computer 20 in communication with thefirst and second VR headsets. The system 10 comprises a memory 22 incommunication with the computer 20. The memory 22 storing an original 3Ddigital representation of the room 17. The computer 20 calculates arendered 3D VR scene 25 for the first VR headset from a point of view ofthe first participant's two eyes, based on the first VR headset'scurrent position and orientation. Anything in the room 17 that isdifferent from the original 3D digital representation of the room 17 isvisible to the first participant 16 in the rendered 3D VR scene 25. Anypart of the room 17 that was present in the original 3D digitalrepresentation of the room 17 becomes visibly replaced by the rendered3D VR scene 25. The computer 20 calculates a rendered 3D VR scene 25 forthe second VR headset from a point of view of the second participant'stwo eyes, based on the second VR headset's current position andorientation. Anything in the room 17 that is different from the original3D digital representation of the room 17 is visible to the secondparticipant 21 in the rendered 3D VR scene 25. Any part of the room 17that was present in the original 3D digital representation of the room17 becomes visibly replaced by the rendered 3D VR scene 25.

The system 10 may include a modem 32 which is in wireless communicationwith the first and second VR headsets. The first headset 14 may send tothe computer 20 at first periodic intervals two camera images capturedby the stereo camera pair 18 of the first headset 14 and at secondperiodic intervals the first headset's position and orientation in thephysical room 17. The second headset 19 maybe send to the computer 20 atthe first periodic intervals two camera images captured by the stereocamera pair 18 of the second headset 19 and at the second periodicintervals the second headset's own position and orientation in thephysical room 17. The first and second periodic intervals of the firstheadset 14 may be different than the first and second periodic intervalsof the second headset 19. The first periodic intervals may be differentfrom the second periodic intervals. Each participant has two eyes whichcomprise a left eye and a right eye, and the computer may calculate therendered VR scene 25 from a point of view of each of the participant'stwo eyes based on each participant's current position and orientation inthe physical room 17.

The computer 20 may compute a distance using the two camera images, asseen at every pixel from each camera to a physical object 33 in thephysical room 17. For each left eye image and right eye image of thecomputed rendered VR scene 25, at every pixel for which a first distancecomputed by the computer 20 is smaller by a threshold DELTA than asecond distance to the nearest point within the stored original 3Ddigital representation of the room 17 as seen from that same pixel, thecomputer 20 may set that same pixel to a color of a corresponding pixelfrom a respective camera in the color camera stereo pair 18 to create aresulting left eye image and right eye image of the rendered VR scene25. Each participant sees the physical object 33 and the physical room17 and also sees a virtual object 4 in the VR scene 25.

The computer 20 may transmit to each VR headset, the resulting left eyeimage and right eye image of the VR scene 25 so anything in the physicalroom 17 that was not already part of original 3D digital representationof the room 17, are visible to each participant and any part of thephysical room 17 that was present during the original 3D scan of thephysical room 17 becomes visibly replaced by the rendered VR scene 25.Each headset may include a set processor 30, as shown in FIG. 1—embodiment 3, and the computer 20 wirelessly transmits back to each VRheadset a silhouette image for each of the participant's right eye andleft eye, which is a first color wherever the camera images from thecameras of the stereo camera pair 18 should be visible, and a secondcolor different from the first color wherever images from the renderedVR scene 25 should be visible. The silhouette image is used by the setprocessor 30 on each VR headset to selectively set each pixel of therendered 3D VR scene 25 to either a corresponding pixel of the cameraimage from the camera wherever the silhouette image is the first coloror the corresponding pixel from the rendered 3D VR scene 25 wherever thesilhouette image is the color.

The present invention pertains to a system 10 for implementing a sharedmixed reality experience to a first participant 16 and a secondparticipant 21 in a same physical room 17, as shown in FIG. 1 ,embodiment 2 and FIG. 3 . The system 10 comprises a first VR headsetworn by the first participant 16 in the physical room 17. The system 10comprises a second VR headset worn by the second participant 21 in thephysical room 17. Each VR headset as a forward-facing color camerastereo pair 18 and a memory 22. The memory 22 storing an original 3Ddigital representation of the physical room 17. The system 10 comprisesa first auxiliary processor 24 adjacent to the first participant 16. Thefirst auxiliary processor 24 in wired connection 6 to the first VRheadset worn by the first participant 16. The system 10 comprises asecond auxiliary processor 26 worn by the second participant 21. Thesecond auxiliary processor 26 in wired connection 6 to the second VRheadset worn by the second participant 21. The memory 22 of the firstheadset 14 worn by the first participant 16 in communication through thewired connection 6 with the first auxiliary processor 24 worn by thefirst participant 16. The memory 22 of the second headset 19 worn by thesecond participant 21 in communication through the wired connection 6with the second auxiliary processor 26 worn by the second participant21. The first auxiliary processor 24 may be worn on a belt or armband ofthe first participant 16, or may be attached to a chair the firstparticipant 16 is sitting in or may be positioned under the chair inwhich the first participant 16 may be sitting. The second auxiliaryprocessor 26 may be worn or positioned the same way by the secondparticipant 21.

The first auxiliary processor 24 worn by the first participant 16calculates a rendered 3D VR scene 25 for the first headset 14 worn bythe first participant 16 from a point of view of the first participant'stwo eyes, based on the first VR headset's current position andorientation. Anything in the room 17 that is different from the original3D digital representation of the room 17 is visible to the firstparticipant 16 in the rendered 3D VR scene 25. Any part of the room 17that was present in the original 3D digital representation of the room17 becomes visibly replaced by the rendered 3D VR scene 25. The secondauxiliary processor 26 worn by the second participant 21 calculates arendered 3D VR scene 25 for the second headset 19 worn by the secondparticipant 21 from a point of view of the second participant's twoeyes, based on the second VR headset's current position and orientation.Anything in the room 17 that is different from the original 3D digitalrepresentation of the room 17 is visible to the second participant 21 inthe rendered 3D VR scene 25. Any part of the room 17 that was present inthe original 3D digital representation of the room 17 becomes visiblyreplaced by the rendered 3D VR scene 25.

The first headset 14 may send to the first auxiliary processor 24 atfirst periodic intervals two camera images captured by the stereo camerapair 18 of the first headset 14 and at second periodic intervals thefirst headset's position and orientation in the physical room 17. Thesecond headset 19 sends to the second auxiliary processor 26 at thefirst periodic intervals two camera images captured by the stereo camerapair 18 of the second headset 19 and at the second periodic intervalsthe second headset's own position and orientation in the physical room17. Each participant has two eyes which comprise a left eye and a righteye, and the first auxiliary processor 24 may calculate the rendered VRscene 25 from a point of view of each of the first participant's twoeyes based on the first participant's current position and orientationin the physical room 17. The second auxiliary processor 26 may calculatethe rendered VR scene 25 from a point of view of each of the secondparticipant's two eyes based on the second participant's currentposition and orientation in the physical room 17.

The first auxiliary processor 24 may compute a distance using the twocamera images of the first head set, as seen at every pixel from eachcamera of the first head set to a physical object 33 in the physicalroom 17. The second auxiliary processor 26 may compute a distance usingthe two camera images of the second head set, as seen at every pixelfrom each camera of the second head set to the physical object 33 in thephysical room 17. For each left eye image and right eye image of thecomputed rendered VR scene 25, at every pixel for which a first distancecomputed by the first auxiliary processor 24 is smaller by a thresholdDELTA than a second distance to the nearest point within the storedoriginal 3D digital representation of the room 17 as seen from that samepixel. The first auxiliary processor 24 sets that same pixel to a colorof a corresponding pixel from a respective camera in the color camerastereo pair 18 of the first head set to create a resulting left eyeimage and right eye image of the rendered VR scene 25.

The first auxiliary processor 24 for the right eye of the firstparticipant 16 may render the right eye image as a texture mapped ontothe rendering of a 3D geometry of the physical room 17 in a depth bufferin the memory 22 and may render a computed stereo depth image as a 3Dsurface into the depth buffer, displaced away from the headset by DELTA.The first auxiliary processor 24 may use the right camera 19 image fromthe right camera 19 as a texture that is mapped onto the 3D surface whenrendering the right eye image where the right eye camera image appearsat pixels where the right camera 19 is seeing something nearer than whatis at a corresponding pixel in the rendered VR scene 25. The rendered VRscene 25 may appear at pixels where the right camera 19 is seeingsomething that is not nearer than what is at the corresponding pixel inthe rendered VR scene 25.

The present invention pertains to a method for implementing a sharedmixed reality experience to participants in a same physical room 17. Themethod comprises the steps of storing an original 3D digitalrepresentation of the room 17 in a memory 22. There is the step ofcalculating with a computer 20 a rendered 3D VR scene 25 for at leastone VR headset worn by a participant of a plurality of headsets worn byparticipants in the room 17 from a point of view of each of theparticipant's two eyes, based on the one VR headset's current positionand orientation. Anything in the room 17 that is different from theoriginal 3D digital representation of the room 17 is visible to theparticipant in the rendered 3D VR scene 25. Any part of the room 17 thatwas present in the original 3D digital representation of the room 17becomes visibly replaced by the rendered 3D VR scene 25. Each VR headsethas a forward-facing color camera stereo pair 18. The computer 20 is incommunication with at least the one VR headset. The memory 22 is incommunication with the computer 20. There is the step of displaying inthe one VR headset the rendered VR scene 25.

The invention is a technique whereby multiple participants who aresharing a VR experience in the same unprepared physical room 17 cancontinue to appear to all other participants as their true physicalselves, rather than appearing to each other as synthetic avatars, andalso so that all participants can see and manipulate selected physicalobjects in the room 17 such that those objects likewise retain theirtrue physical appearance to all participants.

Physical Components

One or more VR headsets, each of which may have a transmitter andreceiver signals, such as data or instructions.

A forward-facing color camera stereo pair 18 incorporated into each VRheadset

An auxiliary depth-from-stereo processor for each VR headset

A means of wireless data connection such as Wifi

A computer 20

User Experience

One or more participants in the same room 17 share a VR experience. Forevery participant, the physical room 17 itself is visually replaced by ashared computer-generated VR world, yet every participant in the room 17continues to see all the other participants in their actual positions inthe room 17. Objects 33, which can be furniture, can be added to theroom 17, and these objects 33 also continue to be visible to allparticipants in their actual positions in the room 17. Everywhere that aparticipant looks, where there is not another person or an added object33, participants will see the shared computer-generated VR world, ratherthan the physical room 17 itself.

Enabling Technology

One or more participants in a room 17 are each wearing a VR headsetwhich incorporates a forward-facing color camera stereo pair 18. Threealternative embodiments are described.

Embodiment 1

Beforehand, the room 17 has been 3D scanned by using any device that isstandard in the field for obtaining 3D scans of rooms, such as aMatterport scanner. The resulting 3D digital representation of thegeometry of the room 17 is stored in a memory 22 in communication withthe computer 20.

Each VR headset is connected wirelessly to the computer 20. At periodicinteractive frame rate intervals, which in one embodiment can be 30times per second, the two images captured by the stereo camera pair 18are sent wirelessly from the headset to the computer 20.

Wherein each headset sends to the computer 20 at periodic intervals twoimages captured by the stereo camera pair 18 and each headset's ownposition and orientation.

Wherein the computer 20 calculates the VR scene 25 from a point of viewof each of the participant's two eyes of each participant based on eachparticipant's current position and orientation in the room 17.

Wherein the computer 20 computes a distance using the two images, asseen at every pixel from each camera to a physical object 33 in thephysical room 17.

Wherein for each of the left eye and right eye images of the computed VRscene at every pixel for which a first distance computed by the computer20 is smaller by a threshold DELTA than a second distance to the nearestpoint within the stored 3D model as seen from that same pixel, thecomputer 20 sets that same pixel to a color of a corresponding pixelfrom a respective camera in the color camera stereo pair 18 to create aresulting left eye image and right eye image of the VR scene 25.

Wherein the computer 20 transmits to each VR headset, the resulting lefteye image and right eye image of the VR scene 25 so anything in thephysical room 17 that was not already part of the original 3D scan ofthe room 17, are visible to each participant and any part of thephysical room 17 that was present during the original 3D scan of thephysical room 17 becomes visibly replaced by the VR scene 25.

The headset itself performs inside out tracking of its own position andorientation using standard techniques, such as have been employed bycommercially available VR headsets including the Quest 2 and the ViveFocus with, for instance, an inside out tracking module. The VR headsetthen wirelessly transmits its position and orientation to the computer20 at periodic intervals, which in one embodiment can be 90 times persecond.

The computer 20 calculates a VR scene 25 from the point of view of eachof the participant's two eyes, based on the VR headset's currentposition and orientation.

In addition, the two stereo images are used by the computer 20 torapidly compute the distance as seen at every pixel from the camera tothe nearest object 33 in the physical room 17 using a depth-from-stereotechnique that is known in the art, such as the technique of [ALightweight Real-time Stereo Depth Estimation Network with DynamicUpsampling Modules. Y Deng, J Xiao, S Z Zhou VISIGRAPP (5: VISAPP),701-710] incorporated by reference herein.

For each of the left eye and right eye images of the computed VR scene25, at every pixel for which the distance computed by stereo depthestimation is smaller by some threshold DELTA than the distance to thenearest point within the stored 3D model as seen from that same pixel,the computer 20 sets that pixel to the color of the corresponding pixelfrom the respective camera in the color camera stereo pair 18. In oneembodiment, DELTA can be 5 cm.

The resulting left and right eye images of the rendered VR scene 25,thus modified by the selective insertion of pixel values from the leftand right cameras, respectively, of the color camera stereo pair 18, arethen transmitted wirelessly from the computer 20 to the VR headset to beviewed by the participant in the VR headset.

In this way, anything in the room 17 that was not already part of theoriginally scanned room 17, such as other people in the room 17 orobjects that have been added to the room 17 subsequent to 3D scanning,are visible to the participant. In contrast, any part of the room 17that was present during the original 3D scan of the room 17 becomesvisibly replaced by the rendered 3D VR scene 25.

Embodiment 2

In an alternate second embodiment, the stereo camera pair 18 isconnected via a wired connection 6 to a dedicated auxiliary processorwhich can be carried or worn by the participant. This auxiliaryprocessor uses the stereo disparity between the two camera images of thecolor camera stereo pair 18 to compute stereo depth. The auxiliaryprocessor then sends the resulting depth image, also via a wiredconnection 6, to a set processor 30 in the VR headset. In thisembodiment, the data for the 3D geometry of the physical room 17 hasbeen preloaded onto the VR headset itself.

In this embodiment the VR headset renders the 3D geometry of thephysical room 17 as seen from the current position and orientation ofthe VR headset in order to create a depth image to use for comparison.

As in the first embodiment, for each of the left eye and right eyeimages of the computed VR scene 25, at every pixel for which thedistance computed by stereo depth estimation is smaller by somethreshold DELTA than the corresponding distance to the nearest pointwithin the stored 3D model as seen from that pixel, the processor on theVR headset sets that pixel to the color of the corresponding pixel fromthe respective camera in the stereo camera pair 18.

One way to implement this is to, for each eye:

(1) render the VR scene 25 from that eye;

(2) project the image of that rendered scene 25 as a texture mapped ontothe rendering of the 3D geometry of the physical room 17 in a depthbuffer;

(3) render the computed stereo depth image as a 3D surface into the samedepth buffer, displaced away from the headset by DELTA;

(4) use the camera image from the left and right cameras, respectively,as a texture that is mapped onto that surface when rendering the resultfor the corresponding eye.

The result will be that the camera image will appear at pixels where thecamera is seeing something nearer than what is at the correspondingpixel in the VR scene 25, and the VR scene 25 will appear at pixelswhere the camera is seeing something that is not nearer than what is atthe corresponding pixel in the VR scene 25.

In this second embodiment, there is no need for either an externalcomputer 20 or a wireless connection. Each VR headset, together with itsassociated color stereo camera pair 18 and dedicated auxiliarydepth-from-stereo processor, functions as a fully self-contained unit,with no need to communicate with any other devices. If there is dynamiccontent being displayed in the VR scene 25, a synchronization signal maybe sent to all the headsets of all the participants, so the participantsall view the VR scene 25 as it is presented at the same time, except foreach participant's unique position in the room 17. This is because allthe participants have the same equipment and software and VR scene 25preloaded into their memories, so VR scene 25 is constantly changing atthe same time and the same way, except for the participant's uniqueposition, which is taken into account by the headset and auxiliaryprocessor of each participant as the VR scene 25 is displayed for eachparticipant.

Embodiment 3

In this embodiment, the VR scene 25 is rendered locally on the headsetof the VR processor, but the determination, for each of theparticipant's two eyes, of which pixels are showing the correspondingimage from the color stereo camera pair 18, versus which pixels areshowing the rendered VR scene 25, is computed on a computer 20 that isconnected via wireless connection to the VR headset, as in embodiment 1.This embodiment has the advantage of not requiring extra localcomputation on the person of each participant of depth from stereo aswould be needed for embodiment 2, while also requiring only a fractionof the wireless bandwidth that would be required to transmit all of thehigh resolution rendered scenes from the computer 20 to all of the VRheadsets as would be the case for embodiment 1. Because of this lowerwireless bandwidth requirement, many more participants can beaccommodated simultaneously in the same room 17, given the same totalwireless bandwidth, than would be the case for embodiment 1.

In this embodiment, the images from the stereo camera pair 18 aretransmitted wirelessly to the computer 20. The computer 20 then performsa depth from stereo computation as in scenario 1, and wirelesslytransmits back to the VR headset a silhouette image for each of theparticipant's two eyes, which in one embodiment is black wherever theimage from the camera should be visible, and white wherever the imagefrom the rendered VR scene 25 should be visible. This silhouette imageis then used by the processor on the VR headset to selectively set eachpixel of the displayed image to either the corresponding pixel of theimage from the camera (wherever the silhouette image is black) or thecorresponding pixel from the rendered VR scene 25 (wherever thesilhouette image is white).

Because the images that need to be transmitted between the computer 20and the VR headset for depth computation are typically much lowerresolution than the images of the final rendered VR scene 25, thismethod requires only a fraction of the total wireless bandwidth requiredto implement embodiment 1.

Alternate Room 17 Model

For any of the above scenarios, a synthetic description of the physicalroom 17 geometry can be used as a substitute for a scan of the actualphysical room 17. To implement this, a 2D shape, such as a rectangle,can be defined as a floor shape which is known to fit entirely withinthe floorplan of the physical room 17. This 2D shape is then extruded inthe vertical direction to create a virtual room 17 volume. The boundaryof that volume can be used to represent the geometry of the room. In oneembodiment the height of the extrusion can be 10 feet.

In scenario 1, this synthetic room geometry description is stored on thecomputer 20. In scenario 2, this synthetic room geometry description isstored on the VR headset's processor.

Although the invention has been described in detail in the foregoingembodiments for the purpose of illustration, it is to be understood thatsuch detail is solely for that purpose and that variations can be madetherein by those skilled in the art without departing from the spiritand scope of the invention except as it may be described by thefollowing claims.

1. A system for implementing a shared mixed reality experience to afirst participant and a second participant in a same physical roomcomprising: a first VR headset having a forward-facing color camerastereo pair worn by the first participant in the physical room; a secondVR headset having a forward-facing color camera stereo pair worn by thesecond participant in the physical room; a computer in communicationwith the first and second VR headsets; and a memory in communicationwith the computer, the memory storing an original 3D digitalrepresentation of the room, the computer calculates a rendered 3D VRscene for the first VR headset from a point of view of the firstparticipant's two eyes, based on the first VR headset's current positionand orientation, anything in the room that is different from theoriginal 3D digital representation of the room is visible to the firstparticipant in the rendered 3D VR scene, any part of the room that waspresent in the original 3D digital representation of the room becomesvisibly replaced by the rendered 3D VR scene, the computer calculates arendered 3D VR scene for the second VR headset from a point of view ofthe second participant's two eyes, based on the second VR headset'scurrent position and orientation, anything in the room that is differentfrom the original 3D digital representation of the room is visible tothe second participant in the rendered 3D VR scene, any part of the roomthat was present in the original 3D digital representation of the roombecomes visibly replaced by the rendered 3D VR scene.
 2. The system ofclaim 1 including a modem which is in wireless communication with thefirst and second VR headsets.
 3. The system of claim 2 wherein the firstheadset sends to the computer at first periodic intervals two cameraimages captured by the stereo camera pair of the first headset and atsecond periodic intervals the first headset's position and orientationin the physical room, and the second headset sends to the computer atthe first periodic intervals two camera images captured by the stereocamera pair of the second headset and at the second periodic intervalsthe second headset's own position and orientation in the physical room.4. The system of claim 3 wherein each participant has two eyes whichcomprise a left eye and a right eye, and the computer calculates therendered VR scene from a point of view of each of the participant's twoeyes based on each participant's current position and orientation in thephysical room.
 5. The system of claim 4 wherein the computer computes adistance using the two camera images, as seen at every pixel from eachcamera to a physical object in the physical room.
 6. The system of claim5 wherein for each left eye image and right eye image of the computedrendered VR scene, at every pixel for which a first distance computed bythe computer is smaller by a threshold DELTA than a second distance tothe nearest point within the stored original 3D digital representationof the room as seen from that same pixel, the computer sets that samepixel to a color of a corresponding pixel from a respective camera inthe color camera stereo pair to create a resulting left eye image andright eye image of the rendered VR scene.
 7. The system of claim 6wherein the computer transmits to each VR headset, the resulting lefteye image and right eye image of the VR scene so anything in thephysical room that was not already part of original 3D digitalrepresentation of the room, are visible to each participant and any partof the physical room that was present during the original 3D scan of thephysical room becomes visibly replaced by the rendered VR scene.
 8. Thesystem of claim 7 wherein the first periodic intervals are differentfrom the second periodic intervals.
 9. The system of claim 8 whereineach headset includes a processor and the computer wirelessly transmitsback to each VR headset a silhouette image for each of the participant'sright eye and left eye, which is a first color wherever the cameraimages from the cameras of the camera image pair should be visible, anda second color different from the first color wherever images from therendered VR scene should be visible, the silhouette image is used by theset processor on each VR headset to selectively set each pixel of therendered 3D scene to either a corresponding pixel of the camera imagefrom the camera of the camera image pair wherever the silhouette imageis the first color or the corresponding pixel from the rendered 3D VRscene wherever the silhouette image is the second color.
 10. The systemof claim 2 wherein each headset includes a set processor and thecomputer wirelessly transmits back to each VR headset a silhouette imagefor each of the participant's right eye and left eye, which is a firstcolor wherever the camera images from the camera should be visible, anda second color different from the first color wherever images from therendered VR scene should be visible, the silhouette image is used by theset processor on each VR headset to selectively set each pixel of therendered 3D scene to either a corresponding pixel of the camera imagefrom the camera wherever the silhouette image is the first color or thecorresponding pixel from the rendered 3D VR scene wherever thesilhouette image is the second color.
 11. A system for implementing ashared mixed reality experience to a first participant and a secondparticipant in a same physical room comprising: a first VR headset wornby the first participant in the physical room; a second VR headset wornby the second participant in the physical room, each VR headset having aforward-facing color camera stereo pair and a memory, the memory storingan original 3D digital representation of the physical room; a firstauxiliary processor adjacent to the first participant; and a secondauxiliary processor worn by the second participant, the first auxiliaryprocessor in wired connection to the first VR headset worn by the firstparticipant, the second auxiliary processor in wired connection to thesecond VR headset worn by the second participant, the memory of thefirst headset worn by the first participant in communication through thewired connection with the first auxiliary processor worn by the firstparticipant, the memory of the second headset worn by the secondparticipant in communication through the wired connection with thesecond auxiliary processor worn by the second participant, the firstauxiliary processor worn by the first participant calculates a rendered3D VR scene for the first headset worn by the first participant from apoint of view of the first participant's two eyes, based on the first VRheadset's current position and orientation, anything in the room that isdifferent from the original 3D digital representation of the room isvisible to the first participant in the rendered 3D VR scene, any partof the room that was present in the original 3D digital representationof the room becomes visibly replaced by the rendered 3D VR scene, thesecond auxiliary processor worn by the second participant calculates arendered 3D VR scene for the second headset worn by the secondparticipant from a point of view of the second participant's two eyes,based on the second VR headset's current position and orientation,anything in the room that is different from the original 3D digitalrepresentation of the room is visible to the second participant in therendered 3D VR scene, any part of the room that was present in theoriginal 3D digital representation of the room becomes visibly replacedby the rendered 3D VR scene.
 12. The system of claim 11 wherein thefirst headset sends to the first auxiliary processor at first periodicintervals two camera images captured by the stereo camera pair of thefirst headset and at second periodic intervals the first headset'sposition and orientation in the physical room, and the second headsetsends to the second auxiliary processor at the first periodic intervalstwo camera images captured by the stereo camera pair of the secondheadset and at the second periodic intervals the second headset's ownposition and orientation in the physical room.
 13. The system of claim12 wherein each participant has two eyes which comprise a left eye and aright eye, and the first auxiliary processor calculates the rendered VRscene from a point of view of each of the first participant's two eyesbased on the first participant's current position and orientation in thephysical room, and the second auxiliary processor calculates therendered VR scene from a point of view of each of the secondparticipant's two eyes based on the second participant's currentposition and orientation in the physical room.
 14. The system of claim13 wherein the first auxiliary processor computes a distance using thetwo camera images of the first head set, as seen at every pixel fromeach camera of the first head set to a physical object in the physicalroom, and the second auxiliary processor computes a distance using thetwo camera images of the second head set, as seen at every pixel fromeach camera of the second head set to the physical object in thephysical room.
 15. The system of claim 14 wherein for each left eyeimage and right eye image of the computed rendered VR scene, at everypixel for which a first distance computed by the first auxiliaryprocessor is smaller by a threshold DELTA than a second distance to thenearest point within the stored original 3D digital representation ofthe room as seen from that pixel, the first auxiliary processor setsthat pixel to a color of a corresponding pixel from a respective camerain the color camera stereo pair of the first head set to create aresulting left eye image and right eye image of the rendered VR scene.16. The system of claim 15 wherein the first auxiliary processor for theright eye of the first participant renders the right eye image as atexture mapped onto the rendering of a 3D geometry of the physical roomin a depth buffer of the memory and renders a computed stereo depthimage as a 3D surface into the depth buffer, displaced away from theheadset by DELTA, and uses a right camera image from a right camera ofthe camera stereo pair of the first head set as a texture that is mappedonto the 3D surface when rendering the right eye image where the righteye camera image appears at pixels where the right camera is seeingsomething nearer than what is at a corresponding pixel in the renderedVR scene, and the rendered VR scene appears at pixels where the rightcamera is seeing something that is not nearer than what is at thecorresponding pixel in the rendered VR scene.
 17. The system of claim 16wherein the original 3D representation of the physical room issynthetic.
 18. The system of claim 10 wherein the original 3Drepresentation of the physical room is synthetic.
 19. A method forimplementing a shared mixed reality experience to participants in a samephysical room comprising: storing an original 3D digital representationof the room in a memory; calculating with a computer a rendered 3d VRscene for at least one VR headset worn by a participant of a pluralityof headsets worn by participants in the room from a point of view ofeach of the participant's two eyes, based on the one VR headset'scurrent position and orientation, anything in the room that is differentfrom the original 3D digital representation of the room is visible tothe participant in the rendered 3D VR scene, any part of the room thatwas present in the original 3D digital representation of the roombecomes visibly replaced by the rendered 3D VR scene, each VR headsethaving a forward facing color camera stereo pair, the computer incommunication with at least the one VR headset, the memory incommunication with the computer; and displaying in the one VR headsetthe rendered VR scene.